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Abstract 

Source coders and channel coders are generally de- 
signed separately without reference to each other. This 
approach is justified by a famous result of Shannons. 
However , there are many situations in practice in 
which the assumptions upon which this result is based 
are violated. Specifically, we examine the situation 
where there is residual redundancy at the source coder 
output. We have previously shown that this residual 
redundancy can be used to provide error correction us- 
ing a Viterbi decoder. In this paper we present the sec- 
ond half of the design; the design of encoders for this 
situation. We show through simulation results that the 
proposed coders consistently outperform conventional 
source-channel coder pairs with gains of up to 12dB at 
high probability of error . 


1 Introduction 

One of Shannon’s many fundamental contributions 
was his result that source coding and channel coding 
can be treated separately without any loss of perfor- 
mance for the overall system [1]. The basic design pro- 
cedure is to select a source encoder which changes the 
source sequence into tid bits followed by a channel en- 
coder which encodes the bits for reliable transmission 
over the channel. However, the separation argument 
no longer holds if either of the following two situations 
occur: 

i. The input to the source decoder is different from 
the output of the source encoder, which happens 
when the link between the source encoder and 
source decoder is no longer error free, or 
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ii. The source coder output contains redundancy. 

Case (i) occurs when the channel coder does not 
achieve zero error probability and case (ii) occurs 
when the source encoder is suboptimal. These two 
situations are common occurrences in practical sys- 
tems where source or channel models are imperfectly 
known, complexity is a serious issue, or significant de- 
lay is not tolerable. Approaches developed for such 
situations are usually grouped under the general head- 
ing of joint source/channel coding. 

Most joint source channel coding approaches 
can be classified in two main categories; (A) ap- 
proaches which entail the modification of the source 
coder/decoder structure to reduce the effect of chan- 
nel errors [2-10], and (B) approaches which examine 
the distribution of bits between the source and chan- 
nel coders [11, 12]. The first set of approaches can 
be divided still further into two classes. One class of 
approaches examines the modification of the overall 
structure [2-5], while the other deals with the modifi- 
cation of the decoding procedure to take advantage of 
the redundancy in the source coder output [6-10]. 

In this paper we present an approach to joint 
source/channel coder design, which belongs to cate- 
gory A, and hence we explore a technique for design- 
ing joint source/channel coders, rather than ways of 
distributing bits between source coders and channel 
coders. We assume that the two nonideal situations 
referred to earlier are present. For a nonideal source 
coder, we use MAP arguments to design a decoder 
which takes advantage of redundancy in the source 
coder output to perform error correction. We then 
use the decoder structure to infer the encoder design. 


2 The Design Criterion 

For a discrete memoryless channel (DMC), let 
the channel input alphabet be denoted by A = 



{ao, ai, . . . , a^-i, }, and the channel input and out- 
put sequences by Y = {yo*yi>-**> 2/L-i} an d Y = 
{Vo, yii • • • , yr-i}i respectively. If A= {Ai} is the set 
of sequences Ai = {<> 1 , 0 , o,* t i, . . . , »»,£,- 1 }, c*i t ktA, then 
the optimum receiver (in the sense of maximizing the 
probability of making a correct decision) maximizes 
P[C] } where 

p[c) = J2 p i c i*wi- 

Ai 

This in turn implies that the optimum receiver max- 
imizes P[C\Y]. When the receiver selects the output 
to be A k} then P[C\Y] = P[Y = A k \Y]. Thus, the 
optimum receiver selects the sequence A k such that 

P[Y = A k \Y] >P[Y = Ai\Y] Vi. 

Lemma 1 

Let yi be the input to a DMC. Given y,- - 1 , Vi 
is conditionally independent of y n -k>k > 1. If 

yo = yo then the optimum receiver ^ selects a^ se- 
quence Ai to maximize TlfS* P{yi\yi-ii Yi) where Y k = 

{yjt, yjfc+i . - - - . yr.-i)- 

The lemma addresses the situation in case (ii), i.e., 
the situation in which the source coder output (which 
is also the channel input sequence) contains redun- 
dancy. Using this lemma, we can design a decoder 
which will take advantage of dependence in the chan- 
nel input sequence. The lemma provides the mathe- 
matical structure for the decoder. The physical struc- 
ture can be easily obtained by examining the quantity 
to be maximized. The optimum decoder maximizes 
P(Y\Y) or equivalently logP(Y|Y), but 

logF(y|y) = £logP(yi|Yi,Vi-i) (!) 

which is similar in form to the path metric of a convo- 
lutional decoder. Error correction using convolutional 
codes is made possible by explicitly limiting the pos- 
sible codeword to codeword transitions, based on the 
previous code input and the coder structure. In this 
case, while there is no structure being imposed by the 
encoder, there is sufficient residual structure in the 
source coder output that can be used for error cor- 
rection. This structure can be quantified in light of 
the Lemma. That is, the structure is reflected in the 
conditional probabilities, and can be utilized via the 
path metric in (1) in a decoder similar in structure 
to a convolutional decoder. However, to implement 
this decoder we need to be able to compute the path 
metric. Unfortunately the quantity i^(y» |Vt*» y*— l) * s 
difficult to estimate. We have therefore used various 


approximations to this quantity with some success. In 
[8, 9] P(y»|Vi,y;-i) is approximated by P(y<|yi, y»-i) 
with excellent results. Other approximations can be 
found in [13]. 

In [9] we showed that the use of the decoder led 
to dramatic improvements under high error rate con- 
ditions. However at low error rates the performance 
improvement was from nonexistent to minimal. This 
is in contrast to standard error correcting approaches, 
in which the greatest performance improvements are 
at low error rates, with a rapid deterioration in per- 
formance at high error rates. In this work we combine 
the two approaches to develop a joint source channel 
codec which provides protection equal to the standard 
channel encoders at low error rates while providing 
significant error protection at high error rates. 

3 Proposed Encoder Structure 

In the conventional error protection approach we in- 
troduce structure in the transmitted bitstream. In the 
approach proposed in [9], we use the residual structure 
in the (generally nonbinary) source coder output se- 
quence. To combine the two approaches, we need to 
introduce additional structure without disturbing the 
structure already present. Because of the nature of the 
decoding approach, a convolutional encoder would be 
most appropriate for introducing structure. However, 
a standard binary convolutional encoder will tend to 
destroy the structure in the source coder output. To 
preserve the residual structure while introducing ad- 
ditional structure we propose to use nonbinary convo- 
lutional encoders (NCE) whose input alphabet is the 
output alphabet of the source coder. 

Let x„, the input to the NCE, be selected from the 
alphabet A = {0, 1,2, and let y n , the output 

alphabet of the NCE, be selected from the alphabet 
S = {0, 1,2, ...,M - 1}. Then, two of the proposed 
NCEs can be described by the following mappings 

1 . M = N 2 ] y n = Nz n -i + x n 

The number of bits required to represent the output 
alphabet using a fixed length code is 

flog,(A/)l = [\og 2 {N 2 ]\ = f21og a (tf)l 
Therefore in terms of rate, this coder is equivalent to a 
rate 1/2 convolutional encoder. The encoder memory 
in bits is 2flog 2 (A01 as each output value depends on 
two input values. 

As an example, consider the situation when N — 4. 
Then A = {0,1, 2, 3} and S = {0, 1,2, .... 15}. Given 



the input sequence x n : 0130211033 and assum- 
ing the encoder is initialized with zeros, the output 
sequence will be y n : 0 1 7 12 2 9 5 4 3 15. 

The encoder memory is four bits. Notice that while 
the encoder output alphabet is of size N 2 , at any given 
instant the encoder can only emit one of N different 
symbols as should be the case for a rate 1/2 convo- 
lutional encoder. For example if y n -i = 0, then y n 
will take on a value from {0, 1,2, (AT - 1)}. In gen- 
eral, given a value for y n _i, Ifn will take on a value 
from {aN,ctN + 1, oeN + 2 ,..., ctN -p N — 1}, where 
a = yn- x(modN). This structure can be used by the 
decoder to provide error protection. The encoder is 
shown in Figure la. 

2. M = iV 3 ; y n = A r 2 z 2n -2 + A^Zn-l + 

The final encoder we consider is equivalent to a rate 
2/3 convolutional coder. Notice that while the input 
output relationship looks similar to a rate 1/3 encoder, 
we generate one output for every tw r o inputs. Thus, 
while the number of bits needed to represent one let- 
ter from the output alphabet is three times the bits 
needed to represent a letter from the input alphabet, 
because two input letters are represented by a single 
output letter, the rate is 2/3. Again, assuming a value 
of 4 for N, the output alphabet is of size 64, and for the 
input sequence used previously, the output sequence 
is y n : 0 52 35 22 49 3. 

The encoder memory is again 6 bits. A block dia- 
gram of the encoder is shown in Figure lb. The rate 
of the encoder can also be inferred from the fact that 
while the encoder output alphabet is of size N 3 , at any 
instant the encoder can transmit one of N 2 (instead 
of N) symbols. Given a value for y n -i, Vn can take on 
a value from the alphabet {yN 2 } yN 2 + 1 } ... } yN 2 + 
(N 2 - 1)} where y = y n -i (modN). 

4 Binary Encoding of the NCE Output 

We will make use of the residual structure in the 
source coder output (which is preserved in the NCE 
output) at the receiver. However, we can also make 
use of this structure in selecting binary codes for the 
NCE output. An intelligent assignment of binary 
codes can improve the error correcting performance 
of the system. Our strategy is to try to maximize the 
Hamming distance between codewords that are likely 
to be mistaken for one another. 

First we obtain a partition of the alphabet based 
on the fact that given a particular value for y n -i» 2 M 
can only take on values from a subset of the full al- 


phabet. To see this, consider the rate 1/2 NCE; then 
the alphabet S can be partitioned into the following 
sub-alphabets: 

S s = (jNJN + 1, .... jN + N- 1) j = 0,1, ...N - 1 

where the encoder will select letters from alphabet 
Sj at time n if j = y n _i(mocLV). Now for each 
sub-alphabet we have to pick N codewords out of 
M (= N 2 ) possible choices. We first pick the sub- 
alphabet containing the most likely letter. The let- 
ters in the sub-alphabet are ordered according to their 
probability of occurrence. We assign a codeword a 
from the list of available codewords to the most prob- 
able symbol. Then, assign the complement of a to 
the next symbol on the list. Therefore the distance 
between the two most likely symbols in the list is 
K = P°g 2 M] bits. We then pick a codeword 6 from 
the list which is at a Hamming distance of K/2 from 
a and assign it and its complement to the next two 
elements on the list. This process is continued with 
the selection of letters that areA/2* away from a at 
the k th step until all letters in the subalphabet have 
a codeword assigned to them. We then pick the sub- 
alphabet that contains the next most likely letter. It is 
assigned the available codeword at maximum distance 
from a. The procedure for assigning codewords within 
the sub-alphabet is then repeated. The assignment for 
a rate 1/2 with N = 4 code is shown in Table 1. 


5 Simulation Results 

The proposed approach was simulated using a two- 
bit DPCM system as the source coder, and the three 
NCE described in section 3. The source used were 
standard test images USC Girl, USC Couple and a 
256x256 portion of Lena. The decoder structure used 
was that of a Viterbi decoder with branch metric log L 

T _ P(yi 1 yi)P(yj 1 Vi-uVi-i) 

L ~ P(Vi) 

where yi denotes the NCE output and yj denotes 
the corrupted channel output. The probabilities 
P(yi I y.*-i>yi-2) were estimated using a training se- 
quence. This requires estimating M N 2 probabilities, 
which were estimated using the USC Girl image. The 
test images were the USC Couple and Lena images. 

The proposed scheme was compared with a con- 
ventional source coder-convolutional coder combina- 
tion. The source coder and source sequence were the 
same in both systems. The convolutional codes se- 
lected were the codes with maximal dj r(t an< l 



same rate and memory characteristics as the proposed 
NCEs from [14]. The performance measure was the 
signal-to-noise-ratio (SNR) defined as 

£u - 2 

SN R= 10 log 10 ~ l ;-. - 

Ew* “ 

where u» is the input to the source encoder and u,- is 
the output of the source decoder. 

The results show consistent improvement in perfor- 
mance for the proposed system. At low probabilities 
of error both systems perform very well. At high prob- 
abilities of error (> 10“ 2 ), however, there is a substan- 
tial improvement in performance when the proposed 
system is used. 

In Figures 2a and 2b we show the results of one 
of the simulations for the rate 1/2 codes. The bi- 
nary assignment of Table 1 was used in the simula- 
tion. Notice the flatness of the performance curve for 
the proposed system. While the proposed system con- 
sistently outperforms the conventional system, it is at 
higher probabilities of error that the differences really 
become significant. At a probability of error of 10 1 
there is almost a 6dB difference in the performance 
of the two systems! This “flattening out” of the per- 
formance curve makes the approach useful for a large 
variety of channel error conditions. 

Similar performance improvements can be seen for 
the rate 2/3 system of the second mapping. The per- 
formance curves are shown in Figure 3. Notice that 
again the proposed system consistently outperforms 
the conventional system. In this case at a probability 
of error of 10“ 1 the performance improvement is more 
than 12dB! In fact, the proposed rate 2/3 system per- 
forms better than the conventional rate 1/2 system. 


6 Conclusion 

If the source and channel coder are designed in a 
“joint” manner, that is the design of each takes into ac- 
count the overall conditions (source as well as channel 
statistics), we can obtain excellent performance over 
a wide range of channel conditions. In this paper we 
have presented one such design. The resulting perfor- 
mance improvement seems to validate this approach. 
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Table 1: Codeword Assignments 


Symbol 

Code 

Symbol 

Code 

0 

0000 

8 

1011 

1 

0011 

9 

0111 

2 

1100 

10 

0100 

3 

1111 

11 

1000 

4 

1110 

12 

0101 

5 

1101 

13 

1001 

6 

0001 

14 

1010 

7 

0010 

15 

0110 


References 

[1] C. E. Shannon. Bell Syst. Tech. J. 27:379-423, 
623-656. 1948. 



(UP) HNS 


Rale 1/2 N'cr.binary ConvoluUcnal Encoder 

(a) 


P.tU 2/3 i'onblnary Ccr.volullcntl Encoder 
(0 


Figure 1. Proposed Nonbinary Convolutional Encoders 
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'igure 2. Performance Comparison of rate 1/2 Systems 
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Figure 3. Performance Comparison of rate 2/3 Systems 
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